Registered reports: an early example and analysis

© 2019 Wiseman et al.The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards.Peer reviewe

Self-other differences in student drinking norms research: the role of impression management, self-deception and measurement methodology

Background: Data-driven student drinking norms interventions are based on reported normative overestimation of the extent and approval of an average student’s drinking. Self-reported differences between personal and perceived normative drinking behaviors and attitudes are taken at face value as evidence of actual levels of overestimation. This study investigates whether commonly used data collection methods and socially desirable responding may inadvertently impede establishing 'objective' drinking norms. Methods: UK students [N=421; 69% female; Mean age 20.22 years (SD = 2.5)] were randomly assigned to one of three versions of a drinking norms questionnaire: The standard multi-target questionnaire assessed respondents' drinking attitudes and behaviors (frequency of consumption, heavy drinking, units on a typical occasion) as well as drinking attitudes and behaviors for an ‘average student’. Two deconstructed versions of this questionnaire assessed identical behaviors and attitudes for participants themselves or an 'average student'. The Balanced Inventory of Desirable Responding was also administered. Results: Students who answered questions about themselves and peers reported more extreme perceived drinking attitudes for the average student compared with those reporting solely on the ‘average student’. Personal and perceived reports of drinking behaviors did not differ between multi- and single-target versions of the questionnaire. Among those who completed the multi-target questionnaire, after controlling for demographics and weekly drinking, socially desirable responding was related positively with the magnitude of difference between students’ own reported behaviors/attitudes and those perceived for the average student. Conclusions: Standard methodological practices and socially desirable responding may be sources of bias in peer norm overestimation research

False positives and other statistical errors in standard analyses of eye movements in reading

In research on eye movements in reading, it is common to analyze a number of canonical dependent measures to study how the effects of a manipulation unfold over time. Although this gives rise to the well-known multiple comparisons problem, i.e. an inflated probability that the null hypothesis is incorrectly rejected (Type I error), it is accepted standard practice not to apply any correction procedures. Instead, there appears to be a widespread belief that corrections are not necessary because the increase in false positives is too small to matter. To our knowledge, no formal argument has ever been presented to justify this assumption. Here, we report a computational investigation of this issue using Monte Carlo simulations. Our results show that, contrary to conventional wisdom, false positives are increased to unacceptable levels when no corrections are applied. Our simulations also show that counter-measures like the Bonferroni correction keep false positives in check while reducing statistical power only moderately. Hence, there is little reason why such corrections should not be made a standard requirement. Further, we discuss three statistical illusions that can arise when statistical power is low, and we show how power can be improved to prevent these illusions. In sum, our work renders a detailed picture of the various types of statistical errors than can occur in studies of reading behavior and we provide concrete guidance about how these errors can be avoided

Using Bayes Factors to evaluate evidence for no effect: examples from the SIPS project

Aims: To illustrate how Bayes Factors are important for determining the effectiveness of interventions. Method: We consider a case where inappropriate conclusions were publicly drawn based on significance testing, namely the SIPS Project (Screening and Intervention Programme for Sensible drinking), a pragmatic, cluster-randomized controlled trial in each of two healthcare settings and in the criminal justice system. We showhow Bayes Factors can disambiguate the non-significant findings from the SIPS Project and thus determine whether the findings represent evidence of absence or absence of evidence. We show how to model the sort of effects that could be expected, and how to check the robustness of the Bayes Factors. Results: The findings from the three SIPS trials taken individually are largely uninformative but, when data from these trials are combined, there is moderate evidence for a null hypothesis (H0) and thus for a lack of effect of brief intervention compared with simple clinical feedback and an alcohol information leaflet (B = 0.24, p = 0.43). Conclusion: Scientists who find non-significant results should suspend judgment – unless they calculate a Bayes Factor to indicate either that there is evidence for a null hypothesis (H0) over a (welljustified) alternative hypothesis (H1), or else that more data are needed

A field experiment on dishonesty: A registered replication of Azar et al. (2013)

This study is a registered replication of a field experiment on dishonesty by Azar et al. (2013). Their main finding was that most customers of an Israeli restaurant did not return excessive change; however, customers who received a higher amount of excessive change returned it more often than people who received a lower amount. Our study, which was conducted on a sample of customers of restaurants in the Czech Republic (N=219), replicated the results of the original study. The high excessive change condition increased the chance of returning the excess change by 21.7 percentage points (17.4 percentage points in the original study). The findings show that the psychological costs of dishonesty can outweigh its financial benefits. We similarly found that repeat customers and women were more likely to return the excessive change than one-time customers and men. The majority (70%) of customers in our sample returned the excessive change. We discuss the importance of field studies and replications of them in the further development of research into dishonest behavior.This study is a registered replication of a field experiment on dishonesty by Azar et al. (2013). Their main finding was that most customers of an Israeli restaurant did not return excessive change; however, customers who received a higher amount of excessive change returned it more often than people who received a lower amount. Our study, which was conducted on a sample of customers of restaurants in the Czech Republic (N=219), replicated the results of the original study. The high excessive change condition increased the chance of returning the excess change by 21.7 percentage points (17.4 percentage points in the original study). The findings show that the psychological costs of dishonesty can outweigh its financial benefits. We similarly found that repeat customers and women were more likely to return the excessive change than one-time customers and men. The majority (70%) of customers in our sample returned the excessive change. We discuss the importance of field studies and replications of them in the further development of research into dishonest behavior